feat: debounce forced-reload trigger endpoints (PER-15248) - #327
feat: debounce forced-reload trigger endpoints (PER-15248)#327dshoen619 wants to merge 6 commits into
Conversation
Replace OpalClient's ungated /policy-updater/trigger and /data-updater/trigger handlers (and route the legacy /update_policy* aliases) through per-updater DebouncedTrigger instances, so an authenticated caller or buggy SDK can no longer amplify full-reload load onto the shared control plane. Coalesces triggers within a configurable window (PDP_TRIGGER_DEBOUNCE_SECONDS, default 10s) and collapses concurrent triggers into the in-flight pull; a failed pull does not consume the window. Response/auth parity with the routes it replaces is preserved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🔍 Vulnerabilities of
|
| digest | sha256:fb3d58cf272e1caa2c503409a0ceb3388342757744514104479261c4e4c33c42 |
| vulnerabilities | |
| platform | linux/amd64 |
| size | 133 MB |
| packages | 248 |
📦 Base Image python:3.13-alpine3.23
| also known as |
|
| digest | sha256:0306b86d5dbbf72135e5e0fcd630005f339b0050b2a2aa5a3946567b14fe0efe |
| vulnerabilities |
Description
Description
Description
Description
Description
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Description
Description
Description
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Description
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Description
|
🔍 Vulnerabilities of
|
| digest | sha256:fb3d58cf272e1caa2c503409a0ceb3388342757744514104479261c4e4c33c42 |
| vulnerabilities | |
| platform | linux/amd64 |
| size | 133 MB |
| packages | 248 |
📦 Base Image python:3.13-alpine3.23
| also known as |
|
| digest | sha256:0306b86d5dbbf72135e5e0fcd630005f339b0050b2a2aa5a3946567b14fe0efe |
| vulnerabilities |
Description
Description
|
There was a problem hiding this comment.
Pull request overview
This PR adds per-updater debouncing/coalescing to the PDP’s forced-reload trigger endpoints to prevent authenticated callers (or buggy SDKs) from repeatedly forcing full control-plane repulls, and replaces OPAL’s trigger-route handlers with PDP-owned gated/debounced equivalents.
Changes:
- Introduces a
DebouncedTriggerutility to coalesce trigger calls within a configurable window and while a reload is in-flight. - Replaces OPAL-mounted
POST /policy-updater/triggerandPOST /data-updater/triggerroutes with PDP-owned,enforce_pdp_token-gated, debounced handlers; legacy aliases share the same debouncers. - Adds
TRIGGER_DEBOUNCE_SECONDSconfiguration (remote-config overridable) and new end-to-end behavior tests for coalescing semantics.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| horizon/debounce.py | Adds the debouncing/coalescing state machine for trigger calls (window + in-flight guard). |
| horizon/pdp.py | Removes OPAL trigger routes and re-registers PDP-owned debounced/gated replacements; wires in per-updater debouncers shared across canonical + legacy aliases. |
| horizon/config.py | Adds TRIGGER_DEBOUNCE_SECONDS (default 10s, 0 disables) to control debounce behavior. |
| horizon/tests/test_trigger_debounce.py | New integration-style tests validating within-window and in-flight coalescing, failure semantics, 0 disables, and canonical/legacy sharing. |
| horizon/tests/test_route_auth_audit.py | Updates regression message to reflect the new route replacement functions. |
| horizon/tests/test_opal_trigger_auth.py | Updates documentation to reflect route replacement (vs dependency injection). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…er-endpoints-to-dampen-reload
…lias) Addresses Copilot review comment on PR #327. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er-endpoints-to-dampen-reload
…R-15248)
Review of the initial implementation found that three of DebouncedTrigger's
documented invariants were false against OPAL's real behaviour, because both
updaters are fire-and-forget underneath: trigger_update_policy is a single put
onto an unbounded asyncio.Queue whose consumer swallows exceptions, and
get_base_policy_data awaits only a config GET before handing the per-entry
fetches to a task pool. So `await run()` returns on DISPATCH, not completion.
Corrections:
- _last_fired -> _last_dispatched, and the "a failed pull does not burn the
window" guarantee is removed. It was never achievable at this layer: a reload
that fails in a background task still consumes the window.
- The in-flight guard no longer claims to cover multi-minute pulls, and is now
unconditional - window_seconds <= 0 disables only the time window, never the
single-flight property.
- Trailing edge: a trigger coalesced by the in-flight guard now causes exactly
one follow-up dispatch, so it is not silently dropped. Capped at two
dispatches per call so a sustained hammer cannot become a reload loop.
Trailing failures are logged and swallowed - the caller executing the re-run
already had its own dispatch succeed and must not be handed someone else's
500. A trigger coalesced into a failed dispatch stays pending instead of
being discarded.
- The /data-updater/trigger docstring claimed a 200 previously meant the fetch
had COMPLETED. It never did; corrected.
Also:
- Routes return {"status": "ok", "triggered": bool} so callers and metrics can
distinguish a dispatch from a coalesce. Documented that `false` is a success
and must not be retried, since retrying re-creates the amplification this
change exists to dampen.
- Handler docstrings were being published as the operation description in the
customer-facing /openapi.json and /scalar explorer, leaking internal notes
including "replaces OpalClient's ungated handler". Replaced with explicit
summary=/description= written for that audience.
- TRIGGER_DEBOUNCE_SECONDS is clamped to [0, 300] and the effective value is
logged at startup. clamp_window coerces defensively rather than raising:
confi.float's cast_from_json is no_cast, so a remote-config override arrives
verbatim, and null or "30" would otherwise abort startup.
- Coalesce logging is INFO on the first suppression per dispatch and DEBUG
thereafter, so the mitigation does not amplify log volume under the exact
hammering it absorbs.
- Config description corrected: the restart requirement comes from remote
config being fetched once at startup, not from the window being read once.
Tests: new test_debounce_unit.py covers DebouncedTrigger directly (burst
collapse, cancellation, trailing edge, clamp_window edges, coalesce logging).
Route-audit now asserts exactly one route per trigger path and PDP ownership,
which the previous last-wins dict lookup could not catch. test_opal_trigger_auth
gets an autouse fixture so per-instance debounce state cannot leak between tests
in that module. 155 passed; ruff check and format clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) Docker Scout began flagging two HIGH CVEs in cryptography 48.0.1 on 2026-08-04, turning docker-scout red on every open PR (#326, #327) and on main. Neither is caused by any code change - the pin has been cryptography>=48.0.1,<49 since #318. CVE-2026-69249 CVSS 8.7 fixed in 49.0.0 CVE-2026-69247 CVSS 8.2 affects >=44.0.0, fixed only in 50.0.0 (Observable Timing Discrepancy) Clearing both requires 50.0.0, so the floor moves past our own <49 major cap. Nothing external bounds cryptography: opal-common 0.9.6 requires it unpinned and its pyjwt[crypto]<3,>=2.4.0 carries no upper bound, so the new <51 cap is ours - it keeps a major out of an image build that has no lockfile, same reasoning as the websockets pin. musllinux_1_2 cp311-abi3 wheels are published for x86_64 and aarch64, so the alpine image keeps installing a prebuilt wheel and still needs no Rust toolchain. No VEX changes: both CVEs are fixable by upgrade, so neither needs a waiver in .docker/scout/pdp-v2.vex.json. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
zeevmoney
left a comment
There was a problem hiding this comment.
Approved — no CRITICAL or HIGH issues found.
Non-blocking:
- MEDIUM
horizon/debounce.py:162— Debounce window is consumed only by successful dispatches, so a failing control plane is unthrottled - MEDIUM
horizon/debounce.py:165— Trailing-edge reload runs inside another caller's request, doubling its worst-case latency - MEDIUM
horizon/pdp.py:590— Window-coalesced triggers are dropped, but the endpoint tells clients not to retry - LOW
horizon/debounce.py:161— In-flight guard has no timeout: one stalled control-plane GET silently no-ops every trigger - LOW
horizon/pdp.py:530— Unparseable remote-config value silently disables debouncing; valid string values log a false warning - LOW
horizon/pdp.py:578— Newtriggeredfield is undeclared in OpenAPI: schema is empty for all four routes - LOW
horizon/pdp.py:600— PR description contradicts the shipped code on the response body and the window semantics
Details are in the inline comments on each line.
| self._pending = False | ||
| try: | ||
| await run() | ||
| self._last_dispatched = time.monotonic() |
There was a problem hiding this comment.
[MEDIUM] Debounce window is consumed only by successful dispatches, so a failing control plane is unthrottled
Problem: self._last_dispatched = time.monotonic() is inside the try at line 160, on the line AFTER await run(). When run() raises, the assignment is skipped, the window is never consumed, and the very next request dispatches again immediately. An anti-amplification control that counts successes instead of attempts provides no damping exactly when the dependency is failing.
This is reachable on the expensive route. _debounced_data_reload (horizon/pdp.py:660) calls DataUpdater.get_base_policy_data, which calls get_policy_data_config — and that function raises ClientError on any non-200 from the control plane (opal_client/data/updater.py:244-249). So under a degraded or 5xx-ing control plane — the scenario config.py:301-302 names as the reason to raise this window — every trigger dispatches a fresh control-plane GET. Only the unconditional in-flight guard remains, which throttles to 1/latency: if the control plane rejects fast (a 503 in a few ms), a single retrying client sustains hundreds of GETs per second from one PDP, multiplied across the fleet.
The exception also propagates out of the handler as a 500 (there is no try/except around await self._debounced_data_reload(...) at horizon/pdp.py:627), which is the one status code SDK and service-mesh retry logic does retry on. So the failure mode actively recruits clients into the retry storm, while a coalesced success is documented as "do not retry".
The module docstring at lines 31-33 asserts "_last_dispatched is a DISPATCH timestamp... There is no 'only on success' guarantee" — true for background failures, but the foreground path implemented here IS only-on-success, and test_trigger_debounce.py:266-270 plus test_debounce_unit.py:381-392 pin that behaviour (assert get_base.await_count == 2 after two consecutive failures).
Suggestion: Record the attempt, not the outcome: stamp _last_dispatched before await run() (or in a finally/except as well), so a failing dispatch still consumes the window. Keep the fast-retry-on-failure behaviour only if it is bounded — e.g. a separate, much shorter failure window (1-2s) — and update the two tests that currently assert the unthrottled behaviour. Also consider mapping the ClientError to a 503 with Retry-After rather than a bare 500, so client retry logic backs off instead of hammering.
Example:
# horizon/debounce.py
self._in_flight = True
self._in_flight_since = time.monotonic()
self._pending = False
try:
await run()
self._log_dispatched()
if self._pending:
await self._run_trailing(run)
return True
finally:
# the window is consumed by the ATTEMPT: a control plane that is
# rejecting must not disable the damping it most needs.
self._last_dispatched = time.monotonic()
self._in_flight = False
self._in_flight_since = None
There was a problem hiding this comment.
Fixed in 2b3be4b — you're right, and I'd add that the direction of the failure made it worse than "no damping": it was anti-damping, since the only condition that consumed the window was the one where damping mattered least.
_last_dispatched is now stamped in a finally, so the attempt consumes the window. I went with the plain version rather than a separate short failure window — a second knob whose only job is to let a failing control plane be hit more often seemed like the wrong thing to add to a mitigation — with one exception: CancelledError does not stamp. A cancelled dispatch was abandoned rather than made, so the control plane was not necessarily asked, and on shutdown there is nobody left to serve a follow-up.
This does not ship alone, though. On its own it converts a hard failure into a lying 200 {"triggered": false} on the retry — the window is consumed by a reload that never happened and, under the old drop semantics, never would. So it lands together with the window-path trailing edge from your pdp.py:590 comment: the coalesce now arms a real retry at window expiry, which is what makes that 200 honest.
Also took the Retry-After suggestion, and went to 502/504 rather than 503. /data-updater/trigger already returns 503 for "data updater is disabled" — a config state where retrying never helps — so overloading it would leave a client unable to tell "back off ten seconds" from "stop forever". 502/504 also matches what this codebase already does for an upstream failure in horizon/enforcer/api.py ("502 indicates server got an error from another server"). Retry-After is the window itself, since the failed attempt just consumed it and anything sooner is provably going to be coalesced.
Both tests you named are rewritten: test_a_run_that_raises_consumes_the_window and test_control_plane_failure_502s_and_consumes_the_window. The module docstring's "there is no 'only on success' guarantee" claim is reworded — it was describing background failures correctly while the foreground path did the opposite.
| self._last_dispatched = time.monotonic() | ||
| self._log_dispatched() | ||
| if self._pending: | ||
| await self._run_trailing(run) |
There was a problem hiding this comment.
[MEDIUM] Trailing-edge reload runs inside another caller's request, doubling its worst-case latency
Problem: _run_trailing is awaited inline at horizon/debounce.py:165, still inside the HTTP request of whichever caller happened to win the dispatch. That caller now pays for a second full reload it never asked for, serving triggers issued by other clients. On the data route the cost is real: run resolves to get_base_policy_data (horizon/pdp.py:664), which awaits _stop_polling_update_tasks() and a data-source config GET whose aiohttp timeout falls back to the 5-minute default — a fact the module docstring itself calls out at horizon/debounce.py:41-45 as the reason the in-flight guard is needed. Doubling that lands against a hard ceiling: the Rust server fronting horizon uses a 60s client timeout (pdp-server/src/config/mod.rs:95) and proxies opaquely (pdp-server/src/api/horizon_fallback.rs:14), so the dispatching caller can be timed out at the proxy for work it did not request, while the reload continues server-side (uvicorn does not cancel the handler on client disconnect). The client then sees a failure for a request that actually succeeded and retries — feeding exactly the load-amplification loop this PR exists to break, and doing it under the degraded-control-plane conditions the feature targets. It also makes the second _stop_polling_update_tasks() cancel the periodic tasks the first run had just created, churning them for no benefit.
Suggestion: Move the trailing run off the request path. Have the debouncer own it as a task (self._trailing_task = asyncio.create_task(...)) with a stored handle so shutdown can cancel it and a completed handle can be cleared, keeping the existing 'at most one trailing run' cap by refusing to schedule when _trailing_task is live. The dispatching caller then returns as soon as its own dispatch is handed off, which is what its 200 already means. If keeping it inline, at minimum bound it (asyncio.wait_for) so one caller's request cannot be extended without limit by other callers' triggers.
Example:
# horizon/debounce.py
await run()
self._last_dispatched = time.monotonic()
self._log_dispatched()
if self._pending and self._trailing_task is None:
# Off the request path: the caller's own dispatch is done, and it
# must not be billed for triggers other callers issued.
self._trailing_task = asyncio.create_task(self._run_trailing(run))
self._trailing_task.add_done_callback(lambda _: setattr(self, "_trailing_task", None))
return True
There was a problem hiding this comment.
Fixed in 2b3be4b, as asyncio.create_task with a stored handle, per your suggestion.
The handle does double duty, which resolved a race I hit on the first attempt: _trailing_task is now part of guard 1 (if self._in_flight or self._trailing_task is not None), not just a cancellation handle. Without that, there is a real gap — the dispatch's finally clears _in_flight, and a trigger arriving before the task gets its first step would start a second concurrent pull. Arming sets the handle before _in_flight is cleared, with no await between, so the two guards hand off atomically. Both finally blocks are now await-free by invariant, and there's a comment saying so, because that's what the whole no-overlap argument rests on.
Lifecycle came out to try/finally plus add_done_callback. The finally gives the deterministic clear-and-chain; the callback covers the one case the coroutine cannot — a task cancelled before its first step never runs its body at all, which would leave the handle set forever and coalesce every future trigger. That's a permanent wedge, so it has its own test. aclose() is wired to app.on_event("shutdown") next to stats_manager.stop_tasks.
One bug I found writing this, worth flagging since it's the shutdown path: cancellation during the trailing run's wait bypassed the inner except CancelledError, so cancelled stayed False and the outer finally armed a replacement task mid-teardown. That's the common aclose() shape — the task is usually waiting out the window, not dispatching — and _pending is still set there. test_aclose_during_the_trailing_wait_does_not_arm_a_replacement fails without the fix (leaves a fresh pending task).
On the second _stop_polling_update_tasks() churn you mention: also improved, and for a slightly different reason — the trailing run now waits out the remainder of the window rather than firing immediately, so back-to-back full pulls are gone entirely.
| "is currently in flight, this call is absorbed into it. Returns 200 either way; " | ||
| "`triggered` reports whether this call started a reload (`true`) or was coalesced " | ||
| "into an existing one (`false`). **`false` is a success, not a failure - do not " | ||
| "retry on it.** It means a reload covering your request is already happening; " |
There was a problem hiding this comment.
[MEDIUM] Window-coalesced triggers are dropped, but the endpoint tells clients not to retry
Problem: The two guards behave differently, and only one of them is honest about it. The in-flight guard arms a trailing edge (self._pending = True, horizon/debounce.py:134) precisely because dropping a coalesced trigger would lose the caller's change. The window guard does not: horizon/debounce.py:140-146 logs and return False without arming anything, and there is no timer anywhere in the module (grep for create_task/call_later/sleep in horizon/debounce.py returns nothing; the class docstring concedes at line 96 that it 'never schedules future work'). So a trigger arriving 1ms after a dispatch completes is discarded outright, and the reload it was 'absorbed into' already read the control plane BEFORE the caller's write landed. Nothing re-runs it: opal-client 0.9.6 has no periodic full-refresh cadence (the PR description itself notes the PDP is pubsub-driven), so the only recovery is another trigger. The shipped, customer-facing OpenAPI description then instructs callers not to do that: 'false is a success, not a failure - do not retry on it. It means a reload covering your request is already happening; retrying only adds load to the control plane.' For the window path no reload is happening at all, and it does not cover the request. A client that follows the documented guidance loses the refresh permanently. The internal docstring makes the same false claim in stronger terms — horizon/debounce.py:83-84 asserts 'staleness is bounded by window_seconds by construction, which is the entire point of the knob' — which is only true if a follow-up eventually fires, and none does. This is also the exact risk PER-15248 called out ('must not debounce so aggressively that a legitimately-needed reload is dropped'); the no-op behaviour itself is sanctioned by the issue, but the guarantee advertised on top of it is not.
Suggestion: Pick one and make code and prose agree. Either (a) arm the trailing edge on the window path too — set _pending at horizon/debounce.py:145 and have the debouncer own a background task that fires at _last_dispatched + window_seconds, which is what makes 'staleness bounded by window_seconds' actually true; or (b) keep the drop and fix both texts: delete 'do not retry on it' and the 'a reload covering your request is already happening' sentence from horizon/pdp.py:589-591 and :613-614, replacing them with something the code backs, e.g. 'a reload dispatched within the last PDP_TRIGGER_DEBOUNCE_SECONDS covers requests made before it started; if you need a reload for a change made after that, retry once the window has elapsed.' Then correct horizon/debounce.py:83-84 to say the window path drops the trigger with no follow-up.
Example:
# horizon/debounce.py — option (a), the version that makes the docstring true
if window_seconds > 0 and self._last_dispatched is not None:
elapsed = time.monotonic() - self._last_dispatched
if elapsed < window_seconds:
self._note_coalesced(...)
# A window-coalesced trigger is NOT covered by the dispatch it
# collapsed into - that one already read the control plane. Schedule
# the follow-up so staleness really is bounded by window_seconds.
self._arm_trailing_timer(run, delay=window_seconds - elapsed)
return False
There was a problem hiding this comment.
Fixed in 2b3be4b — took option (a), so the docstring's claim becomes true rather than getting deleted.
What decided it was running the numbers on load, since "(a) costs more control-plane traffic" was the only real argument for (b). It doesn't. The timer fires at _last_dispatched + window and goes through trigger() itself, so its dispatch is the next window's leading dispatch:
| (b) drop | (a) trailing timer | |
|---|---|---|
| sustained hammer | 1 leading + 1 in-flight trailing = 2 per window | 1 timer-leading + 1 in-flight trailing = 2 per window |
| burst of 3, then silence | 1 total (2 triggers lost forever) | 2 total |
| idle | 0 | 0 |
Under the abuse case the mitigation exists for, the two are identical — the hammer already guarantees a dispatch at every window boundary; all (a) changes is who initiates it. The only cost is one extra reload per burst-then-silence episode, which is exactly the case where (b) silently loses a legitimate refresh. Given PER-15248's "must not debounce so aggressively that a legitimately-needed reload is dropped", that trade only goes one way. The background-task machinery was needed for your debounce.py:165 comment regardless, so it was mostly sunk cost.
Termination was the thing I had to be careful about — an unconditionally self-re-arming timer would be a standing 1-reload-per-window load on the control plane with nobody asking for anything, strictly worse than the status quo. The invariant: _pending is written True in exactly one place (trigger(), by an inbound caller) and nothing in the trailing path sets it, so chain length is bounded by the number of real triggers and stops one run after they do. test_trailing_edge_chains_while_triggers_keep_arriving_then_stops pins both halves.
Prose rewritten anyway, because even under (a) "a reload covering your request is already happening" was wrong — it's scheduled, not happening. Both route descriptions now say the follow-up "begins after your call, within PDP_TRIGGER_DEBOUNCE_SECONDS", and both gained the best-effort caveat that only the data route had. The class docstring's "this class never schedules future work" and "capping the work at two dispatches per call" are gone; the config description is updated too.
| # the next dispatch clears it right here, at the point where it actually serves it. | ||
| self._pending = False | ||
| try: | ||
| await run() |
There was a problem hiding this comment.
[LOW] In-flight guard has no timeout: one stalled control-plane GET silently no-ops every trigger
Problem: await run() is unbounded, and _in_flight stays True for its entire duration. Every trigger arriving in that window returns 200 with {"triggered": false} and, per the published description (horizon/pdp.py:590), the caller is told a reload "covering your request is already happening" and not to retry.
For the data route, run() blocks on get_policy_data_config, which builds ClientSession(headers=..., trust_env=True) with no timeout argument (opal_client/data/updater.py:240-243) and therefore inherits aiohttp's 5-minute default. The module docstring acknowledges this at lines 34-37. The consequence is not acknowledged: a single stalled GET turns the PDP's only forced-reload escape hatch into a silent no-op for up to five minutes, with no distinguishing status code, no header, and no health signal — and the trailing edge then serves all of those absorbed triggers with exactly one catch-up run.
This is new shared state. Before this PR each trigger request performed its own reload, so one stalled request could not convert other callers' requests into no-ops. It is bounded delay rather than loss (the trailing edge does fire), which is why I have not rated it higher, but the window is long, it is unobservable to the caller, and it coincides with exactly the degraded-control-plane condition the feature targets.
Suggestion: Bound the in-flight duration so the guard cannot outlive a plausible reload: wrap the dispatch in asyncio.wait_for(run(), timeout=<a few seconds>), or pass an explicit aiohttp.ClientTimeout upstream. Independently, make the state observable — surface in_flight / in_flight_age in the response body or a debug field, and reconsider the "do not retry" wording for the in-flight case specifically, since a stalled reload is precisely the case where a caller retrying later is correct.
Example:
# horizon/debounce.py
MAX_DISPATCH_SECONDS: float = 30.0
...
try:
await asyncio.wait_for(run(), timeout=MAX_DISPATCH_SECONDS)
self._last_dispatched = time.monotonic()
...
except TimeoutError:
logger.error(
"{} reload dispatch exceeded {:g}s; releasing the in-flight guard so "
"subsequent triggers are not silently absorbed.",
self._name, MAX_DISPATCH_SECONDS,
)
raise
There was a problem hiding this comment.
Half-taken, and I'd like you to sanity-check the half I declined.
Declined: asyncio.wait_for(run(), ...). It cancels the inner coroutine, and for the data route that lands in the middle of a destructive sequence. DataUpdater.get_base_policy_data:
:268 await self._stop_polling_update_tasks() # cancels + clears EVERY periodic poller
:271 sources_config = await self.get_policy_data_config(...) # the unbounded GET
:296 self._polling_update_tasks.append(asyncio.create_task(...)) # the ONLY place they are recreated
The stall is the GET, so a 30s timeout lands squarely between 268 and 296 and permanently kills every periodic_update_interval data source. Nothing reschedules them except another successful get_base_policy_data — an OPAL reconnect, or a later trigger that gets through — and it's silent: no error, no log, just data that stops refreshing. Weighed against what it buys: the wedge is already bounded at aiohttp's 5-minute default and self-heals. Swapping a bounded, self-healing coalescing stall for open-ended silent staleness of every periodic source looked like the wrong side of the trade, especially under the degraded-control-plane conditions where both are most likely.
(wait_for also doesn't hard-bound it — it awaits the inner cancellation to complete, so an uncooperative coroutine holds the guard past the timeout anyway.)
Taken: the observability half, which I think was the actual complaint. MAX_DISPATCH_SECONDS = 30.0 is now a detection threshold rather than a cancellation one: past it, every coalesce logs at ERROR ("the control plane looks stalled and forced reloads are being absorbed, not served") instead of being buried at DEBUG. 30s is comfortable for detection — the awaited work is a task-cancel gather plus one small JSON GET.
The "silently absorbed" part is also materially better now for a reason that isn't in your comment: with the trailing edge armed on both guards, triggers absorbed during a stall set _pending and are served when it clears, rather than being dropped. So it's bounded delay rather than loss, which is closer to what the "do not retry" wording promises.
If you'd still like a hard release, the safe shape is wait_for(shield(task), ...) — releases the guard, leaves the dispatch running, worst case duplicates pollers once and self-heals. Happy to add it as a follow-up; I didn't want to smuggle it into a review-fix commit.
| # overridable and is clamped to [0, MAX_DEBOUNCE_SECONDS], so a fat-fingered override | ||
| # should be visible at startup rather than silently reinterpreted. | ||
| effective_window = clamp_window(sidecar_config.TRIGGER_DEBOUNCE_SECONDS) | ||
| if effective_window != sidecar_config.TRIGGER_DEBOUNCE_SECONDS: |
There was a problem hiding this comment.
[LOW] Unparseable remote-config value silently disables debouncing; valid string values log a false warning
Problem: Two problems in the same handful of lines, both stemming from comparing and coercing an untyped remote-config value. (1) Fail-open direction: clamp_window returns 0.0 on TypeError/ValueError (horizon/debounce.py:66-67), so a remote-config null or a typo such as "tem" turns the amplification mitigation off entirely rather than falling back to the declared 10.0 default. For a control the whole PR exists to provide, 'uninterpretable' should degrade to the safe default, not to disabled — the same reasoning the author applies at horizon/debounce.py:47-49 to justify clamping a fat-fingered large value instead of honouring it. (2) False warning: the startup check compares a float against the raw attribute, and confi's cast_from_json is no_cast for remote overrides — a fact the clamp_window docstring states explicitly at horizon/debounce.py:57-59. So a perfectly valid override delivered as the JSON string "30" yields 30.0 != "30" and logs PDP_TRIGGER_DEBOUNCE_SECONDS=30 is out of range; clamped to 30s (max 300s), which is false on both counts and directly undercuts the stated purpose of the branch ('a fat-fingered override should be visible at startup rather than silently reinterpreted', horizon/pdp.py:526-528). Neither case is tested: test_clamp_window parametrises floats only.
Suggestion: Give clamp_window an explicit fallback (clamp_window(value, default=10.0)) returning the default rather than 0.0 when the value is uninterpretable, and keep 0.0 reserved for an explicit, parseable 0. For the warning, compare against the coerced value — compute configured = float(...) where possible and warn only when the clamp actually changed a numeric value, so a type-only difference stays silent. Extend the test_clamp_window parametrisation with the shapes the helper is documented to handle: "30", None, "", and a non-numeric string.
Example:
# horizon/pdp.py
configured = sidecar_config.TRIGGER_DEBOUNCE_SECONDS
effective_window = clamp_window(configured)
try:
numeric = float(configured)
except (TypeError, ValueError):
numeric = None
if numeric is None or effective_window != numeric:
logger.warning("PDP_TRIGGER_DEBOUNCE_SECONDS={!r} is not a usable window; using {:g}s (max {:g}s).", configured, effective_window, MAX_DEBOUNCE_SECONDS)
There was a problem hiding this comment.
Both fixed in 2b3be4b, essentially as you wrote it.
Fail-open → fail-safe. resolve_window(value, default=DEFAULT_DEBOUNCE_SECONDS) returns (window, problem) where problem is "unparseable" | "clamped" | None; clamp_window is a thin wrapper for the per-request path. Anything uninterpretable — None, "tem", non-finite, and negatives — falls back to the default, so the mitigation stays on. I folded negatives in with your reasoning: -1 is a typo, not a request to disable. Only an explicit, parseable 0 disables the window now, which also makes a remote null distinguishable from a deliberate 0. Annotation is Any, not float — the old float hint contradicted the function's own docstring about what actually reaches it.
False warning. The comparison is now float-to-float inside resolve_window, so "30" reports no problem at all. The two cases got split, since they warrant different severities: unparseable logs at ERROR ("is not a usable window; falling back to the default 10s. Forced-reload trigger debouncing REMAINS ENABLED"), clamped stays at WARNING with an accurate "allowed 0-300s". Both use {!r} so 'tem' and 10.0 are distinguishable in logs.
Single-sourcing: DEFAULT_DEBOUNCE_SECONDS lives in debounce.py beside MAX_DEBOUNCE_SECONDS, and config.py imports it for the confi.float default — so the value the setting declares and the value the clamp substitutes cannot drift. Direction matters and I left a comment saying so: debounce.py must never import config.py back, and doesn't need to, since trigger() takes the window as a parameter.
Tests: test_clamp_window is extended with "30", None, "", "tem" and the negative, and there's a new test_resolve_window_reports_why_the_value_changed that pins the "30"-is-not-a-clamp case specifically, since that's the regression.
| # FastAPI publishes a handler docstring as the operation `description` in the | ||
| # customer-facing /openapi.json and /scalar explorer. The explicit summary=/description= | ||
| # below win over the docstring and are written for that audience. | ||
| @app.post( |
There was a problem hiding this comment.
[LOW] New triggered field is undeclared in OpenAPI: schema is empty for all four routes
Problem: The PR goes out of its way to write customer-facing OpenAPI copy (the comment at horizon/pdp.py:574-577 says the summary=/description= are 'written for that audience', i.e. /openapi.json and /scalar), and that copy instructs clients to branch on a brand-new response field: 'triggered reports whether this call started a reload (true) or was coalesced into an existing one (false)'. But neither replacement route declares a response_model, and neither handler carries a return type annotation (async def trigger_policy_update(): at :594, async def trigger_data_update(): at :620). rg response_model horizon/pdp.py returns nothing. With no response_model and no return annotation, FastAPI emits the default 200 response with an empty JSON schema, so the published contract prose tells integrators to read a field the machine-readable schema does not describe - and code generators / typed SDKs get nothing to bind to. The same applies to the two legacy aliases at :543 and :554, though those are include_in_schema=False so only the two canonical routes are customer-visible. This is the one externally visible shape change in the PR, and it is the half that did not make it into the schema.
Suggestion: Declare the shape once and reuse it on both canonical routes, e.g. a small class TriggerResponse(BaseModel): status: str; triggered: bool and response_model=TriggerResponse on the two @app.post decorators (or simply annotate the handlers -> TriggerResponse and return it). That makes triggered appear in /openapi.json and /scalar alongside the prose that already tells clients to use it, and it pins the body shape against future drift.
Example:
class TriggerResponse(BaseModel):
status: str
triggered: bool
@app.post(
"/policy-updater/trigger",
status_code=status.HTTP_200_OK,
response_model=TriggerResponse,
tags=["Policy Updater"],
dependencies=[Depends(enforce_pdp_token)],
summary="Trigger a full policy reload",
description=(...),
)
async def trigger_policy_update() -> TriggerResponse:
logger.info("triggered policy update from api")
return TriggerResponse(status="ok", triggered=await self._debounced_policy_reload())
There was a problem hiding this comment.
Fixed in 2b3be4b. TriggerResponse (pydantic v1, matching the <2 pin) with status: Literal["ok"] and triggered: bool, response_model= plus return annotations on all four routes.
Verified against the generated schema rather than assuming:
/policy-updater/trigger -> responses: ['200']
200 schema: {"$ref": "#/components/schemas/TriggerResponse"}
/data-updater/trigger -> responses: ['200', '502', '503', '504']
200 schema: {"$ref": "#/components/schemas/TriggerResponse"}
Added responses={502, 503, 504} on the data route too — the prose described a 503 the schema never mentioned, and the new gateway errors would have had the same problem. test_openapi_declares_the_trigger_response_shape pins all of it so it can't drift back.
Put it on the two legacy aliases as well. Not for docs — include_in_schema=False keeps them out, and the test asserts they stay out — but because response_model also validates at runtime, and that's what keeps all four bodies in lockstep given they share one debouncer.
One thing your comment made me catch on myself: I first wrote the rationale as the model's docstring, and pydantic publishes that as the schema description, so /scalar was showing "Declared as a response_model (rather than left as a bare dict)…" to customers — the exact trap the note at pdp.py:574 warns about for handlers. Moved to a comment; the docstring is now "The result of a forced-reload trigger."
On placement: I put it at module level in pdp.py rather than horizon/system/schemas.py, since that package is the /version + /_exit system router and these aren't system routes. horizon/connectivity/api.py (models defined in the module that mounts the routes) felt like the closer precedent. Easy to move if you'd rather it lived in a schemas module.
| # replaces, so a 200 means the same thing it always did. | ||
| logger.info("triggered policy update from api") | ||
| triggered = await self._debounced_policy_reload() | ||
| return {"status": "ok", "triggered": triggered} |
There was a problem hiding this comment.
[LOW] PR description contradicts the shipped code on the response body and the window semantics
Problem: The description is stale against the head commit in ways that matter for review and release notes, and it makes safety claims the code does not support. It states 'Body stays exactly {"status": "ok"} so SDKs never error-spiral' — but all four handlers now return an extra triggered field, which is the one externally visible shape change in this PR. It states '_last_fired recorded only on success, so a failed pull doesn't burn the window' — there is no _last_fired; the field is _last_dispatched, it records a dispatch, and a test exists specifically to pin the opposite ('there is no "only on success" guarantee to be had at this layer'). It cites a review pass 'confirming ... the success-only timestamp holds', a property that no longer exists. It still carries a '
Suggestion: Rewrite the description against the head commit: state that the body gains triggered, that _last_dispatched is a dispatch timestamp with no success guarantee (both updaters being fire-and-forget), and drop the draft/rebase section and the stale review-pass claim. If a changelog or version bump is expected for a customer-visible body change on four endpoints, add it here.
There was a problem hiding this comment.
Rewritten against the head commit — you were right that it had drifted into actively misleading territory, and the {"status": "ok"} claim was the worst of it, since it told a reader the exact opposite of the one client-visible change.
Fixed: the body section now states that triggered is added (and that status is unchanged, which is the part that matters for SDK compatibility); the _last_fired / "only on success" paragraph is gone and replaced with the actual _last_dispatched attempt semantics; the stale review-pass claim and the "
Also took the release-notes point. There's now a dedicated "Client-visible changes" section listing the three externally visible items rather than leaving them scattered through the prose:
- the body gains
triggered(statusunchanged), - a failed control-plane fetch answers 502/504 +
Retry-Afterinstead of a bare 500, - a retry inside the window after a failure returns
200 {"triggered": false}rather than a second 500.
(3) is new in this round and is the one I'd most want a second opinion on before release — it's the customer-facing consequence of making a failed dispatch consume the window, discussed on your debounce.py:162 thread.
Zeev's review raised 7 non-blocking findings. All are real; six are fixed as
suggested, one is fixed differently because the suggested remedy regresses.
The three MEDIUMs are coupled and land together:
* A failed dispatch now consumes the debounce window. `_last_dispatched` is
stamped in a `finally` rather than after `await run()` succeeds, so a control
plane returning 5xx (get_policy_data_config raises ClientError on any non-200)
is damped instead of admitting a fresh GET per request. Cancellation is the one
exception: the attempt was abandoned, not made.
* A window-coalesced trigger is no longer dropped. Both guards now arm a trailing
run that fires at window expiry, so "staleness is bounded by window_seconds" is
a guarantee rather than a comment. Dropping it lost the refresh permanently -
the PDP is pubsub-driven with no periodic full-refresh - which PER-15248
explicitly forbids. Shipping the window change without this would turn a hard
failure into a 200 {"triggered": false} for a reload that never happens.
* The trailing run moved off the request path into a task. It used to be awaited
inline by whichever caller won the dispatch, billing that caller for a second
full reload it never asked for, against the 60s client timeout of the Rust
server that fronts horizon. The chain terminates: `_pending` is written by
`trigger()` alone, so chain length is bounded by real triggers.
Not taken: wrapping `run()` in `asyncio.wait_for` to bound the in-flight guard.
`get_base_policy_data` tears down every periodic poller (updater.py:268) before
the unbounded config GET and only recreates them at the end (:296), so a timeout
landing on the stalled GET would kill periodic data updates outright, with no
error and no recovery short of an OPAL reconnect - open-ended silent staleness in
place of a bounded, self-healing stall. The stall is made observable instead:
past MAX_DISPATCH_SECONDS every coalesce logs at ERROR.
Remaining findings:
* clamp_window fails safe. An uninterpretable remote override (null, "tem", a
negative) now falls back to the default instead of 0, so a control-plane typo
can no longer switch the mitigation off fleet-wide; only an explicit, parseable
0 disables it. resolve_window reports why a value changed, which kills the false
"out of range; clamped" warning a valid JSON-string override used to trigger
(confi's cast_from_json is no_cast). The default is single-sourced in
debounce.py; config.py imports it (one-way edge).
* TriggerResponse is declared as a response_model on all four routes, so
`triggered` appears in /openapi.json instead of an empty 200 schema, and the
data route documents its 502/503/504. Runtime validation keeps all four bodies
in lockstep. Its docstring is customer-facing; rationale lives in comments.
* A failed control-plane fetch answers 502 (504 on timeout) with Retry-After,
not a bare 500 - the one code every SDK and mesh retries. 502/504 matches
horizon/enforcer/api.py and keeps 503 meaning "data updater disabled", a
permanent config state a client must be able to tell apart. Retry-After is the
window, since the failed attempt just consumed it.
Also fixed a bug found reviewing this change: cancellation during the trailing
run's wait bypassed the inner handler, so aclose() - which usually finds the task
waiting, with `_pending` still set - armed a replacement task mid-teardown. Test
added; it fails without the fix.
183 passed; ruff check and ruff format clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What & why
Closes PER-15248.
Even with auth enforced (PER-15244/45/46), a valid-token caller or a buggy SDK can hammer the forced-reload trigger endpoints, each of which forces a full re-pull from the shared control plane — a load-amplification vector across the fleet. This adds a small per-updater debounce so redundant/concurrent forced reloads coalesce instead of amplifying.
There are four amplifying routes, not two (the issue lists two): the OPAL-mounted
POST /policy-updater/triggerandPOST /data-updater/trigger, plus the PDP's legacy aliasesPOST /update_policyandPOST /update_policy_data, which call the same two updater methods directly. All four are now debounced.How
horizon/debounce.py—DebouncedTrigger: per-updater coalescing on a monotonic clock. Two guards: an in-flight guard that collapses concurrent triggers into the running dispatch regardless of the window, and a window guard for triggers arriving withinPDP_TRIGGER_DEBOUNCE_SECONDSof the last one.DataUpdater.on_connectonly re-fetches on websocket reconnect), and PER-15248 explicitly forbids debouncing so aggressively that a needed reload is dropped. The chain terminates —_pendingis written only by an inbound trigger, so it stops one run after triggers stop.aclose()._last_dispatchedrecords the ATTEMPT, not the outcome — stamped in afinally, so a dispatch that raises still consumes the window.get_policy_data_configraisesClientErroron any non-200, so a window consumed only by successes would switch the mitigation off in exactly the degraded-control-plane conditions it exists for. Cancellation is the one exception: the attempt was abandoned rather than made.horizon/pdp.py: the OPAL-mounted handlers are closures we can't intercept, and a FastAPI dependency can't short-circuit to a200no-op — so we remove the two OPAL routes and re-register PDP-owned,enforce_pdp_token-gated, debounced replacements at the same paths (fail-loud if a path is missing). The two legacy aliases share the same per-updater debouncers, so an alternating hammer still coalesces.horizon/config.py:PDP_TRIGGER_DEBOUNCE_SECONDS(default10.0;0disables; clamped to 300s; remote-config overridable fleet-wide, so ops can raise it to 30–60s under a degraded control plane without a release). Uninterpretable values fail safe to the default, not to disabled.Client-visible changes
Worth calling out explicitly for release notes — these are the externally visible parts:
{"status": "ok", "triggered": <bool>}.statusis unchanged and always"ok", so SDKs that only read it are unaffected.triggeredis declared in/openapi.jsonvia aTriggerResponseresponse model./data-updater/triggeranswers502(or504on timeout) withRetry-After, where it previously escaped as a bare500with no body. 500 is the one status every SDK and service mesh retries, so the old failure mode recruited clients into a retry storm against an already-degraded control plane.503deliberately keeps its existing, distinct meaning on this route — "the data updater is disabled", a config state where retrying never helps — so a client can tell "back off" from "stop".200 {"triggered": false}rather than a second500, because the failed attempt consumed the window. The trailing run retries it at window expiry.Decisions & scope
/kong(issue lists it as optional):/kongresolves against the local OPA cache (horizon/enforcer/api.py), so hammering it loads only that one PDP — no control-plane amplification, which is the entire threat model here. It also needs a different control (a genuine per-request rate limit), not a debounce that would return stale authz decisions.run()was considered and rejected:get_base_policy_datatears down everyperiodic_update_intervalpoller (opal_client/data/updater.py:268) before the unbounded config GET, and only recreates them at the very end (:296). A timeout landing on the stalled GET — which is exactly where it would land — would leave every periodic data source permanently dead, silently, until an OPAL reconnect. That trades a bounded (aiohttp's 5-minute default), self-healing stall for open-ended silent staleness. The stall is made observable instead: pastMAX_DISPATCH_SECONDSevery coalesce logs at ERROR, and the triggers absorbed during it are served by the trailing run rather than lost.Tests
horizon/tests/test_debounce_unit.py(45 tests) drives the state machine directly with a fake clock and a fake sleep, covering both guards, the trailing edge and its chain termination, window fail-safe parsing, failure and cancellation paths, task-handle lifecycle (including a task cancelled before its first step),aclose(), and log-level behaviour.horizon/tests/test_trigger_debounce.py(16 tests) covers the same behaviour end-to-end through the real app, plus the published OpenAPI shape and the 502/503/504 mapping.Full suite green (
183 passed), stable over repeated runs;ruff check+ruff format --checkclean.🤖 Generated with Claude Code